Embodied music listening and making in context-aware mobile applications: the EU-ICT SAME Project

Music making and listening are a clear example of a human activity that is above all interactive and social, two big challenges for novel HCI paradigms. Nowadays, however, listening to music is usually still a passive, non-interactive experience. Quoting John Sloboda " In highly industrialized societies, we listen to more music, but we make less " [1]. Even modern devices do not allow for interactive user participation. This can be considered a degradation of traditional listening experience, in which the public can interact in many ways with performers. Recently, many research efforts in the HCI and Sound and Music Computing (SMC, [2]) communities have been devoted to the development of techniques enabling an active, participative, personalized experience of sound and music content. These include innovative intelligent real-time content processing on music signals, new paradigms for natural multimodal interfaces, new devices, context-aware music content processing, and experience of music in cooperative social environments. With active experience and active listening we mean that listeners are enabled to interactively operate on music content through their movement and gesture, by modifying and molding it in real-time while listening. Active listening is the basic concept for a novel generation of interactive music systems [3], which are particularly addressed to a general public of beginners, naïve and inexperienced users, rather than to professional musicians and composers. A particularly relevant aspect of active listening is its social, collaborative implication: active listening paradigms enable a social, collaborative, and context aware experience of music, allowing listeners to cooperate in the real-time manipulation and recreation of music content. Examples of the active listening paradigm are emerging. The Orchestra Explorer [4] enables users to explore a space populated by virtual instruments. Mappe per Affetti Erranti [5] introduces multiple levels of navigation: from navigation in a physical space up to emotional spaces populated by different expressive performances of the same music piece. Users can navigate such spaces by their expressive movement and gesture. Mappe per Affetti Erranti also addresses fruition by multiple users encouraging social behavior. The Mobile Phone Orchestra [6] and virtual air guitars [7, 8] are examples of gesture-based mobile musical instruments. They can be made easier to play than conventional musical instruments, because user's gestures can be interpreted by the computer to produce the desired output sound. This paper will discuss the research challenges involved in active listening of sound and music content in shared and distributed collaborative environments, …